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Preface 


Two and a half years ago | gave a series of beginning and advanced workshops in programmed 
instruction at the Center for Programed Instruction. At that time there was no useful 

published programmed material for training programmers. It was always a source of embarrass- 
ment to be asked "Well, if programming is so good, why don't you use it to teach programming?" 
The workshops provided a chance to do at least part of the job. 


Among the most difficult lessons for the trainee programmer is how to be sensitive to the 
feedback he received from subjects during the first tryouts of a self-instructional unit. 

This appeared to be a challenging task to analyze and program. It proved to be especially 
interesting because our analysis led to the conclusion that a programmed audio component 
was needed to simulate some of the situations in which the trainee programmer would find 
himself. The last two exercises in this program require that the trainee programmer listen to 
a tape and critique the interaction he hears. With this experience behind him, he is then 
able to tape record his own interaction with test subjects and critique himself as he has 
practiced doing with the tape. This procedure -- if he uses it -- enables the programmer to 
continue to learn testing technique long after he has exhausted the resources of this program. 


One satisfaction of the instructional systems designer is to have created a system which 
teaches better than he could do in person given the same amount of time. 


Robert E. Horn 
September, 1965 


HOW TO USE THIS PROGRAM 


This program of self-instruction for programmer-trainees has been designed to be 
used as a part of a course in programming or instructional systems design. It is 
assumed that the person taking the program has had some exposure to the use of 
programs and to the writing of sequences before he begins to use "Developmental 
Testing." 


The program is composed of three parts: 


1. The Frame Book. 

2. The Panel Book. 

3. An Audio-Tape recording (which can be used on any standard 
tape recorder). 


The student should begin by using the Frame Book at page one. Instructions from 
time to time in this book will direct his attention to the Panel Book and to the 
Audio-Tape recording. 
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DEVELOPMENTAL TESTING SELF-TEST (PRE-TEST) 


This self-test is a way of establishing what you already may know about testing 
instructional programs. You will, thus, be able to compare what you now know 
with what you have learned from this program. Don't worry if you aren't able 
to do too well on this pre-test. 


DIRECTIONS 


Prepare a checklist or outline of the important things to do and not to do in 
developmental testing (trying out programmed instruction with individual 
students) . 


Write the checklist as if you were making notes to explain the procedure to 
someone else. Be sure to include what you would do before, during, and 
after the tryout. 


Prepare your outline on a separate sheet of paper. Do not write on this book. 
At the top of each sheet, write your name and today's date. 


\u/ Here are some suggested categories for your outline. You do not need to follow them 
exactly if you feel some other classification makes more sense for your notes. 


l. Trainee Population (Test Subjects) 
A. Should be members of the potential student population. 
B. 
ee 


etc, 
Il. Room (Environment for Testing) 
A. 
B. 
etc, 
Ill. | Equipment and Supplies needed. 


IV. Instructions Prior to Test. 


V. The Program Tester's Job. 


a 


WHEN YOU HAVE COMPLETED YOUR CHECKLIST, PUT IT ASIDE AND-TURN TO THE 
‘ws NEXT PAGE. 


WHAT THIS UNIT IS ABOUT 


ee ee 


There are two kinds of tests which can be performed on instructional materials before they are 
put into use. 


1. Developmental Testing (or Student Tryouts), the purpose of which is to obtain information 
for revising the materials. In this kind of test, the materials are given to a few students 
rather than a large number. The students are carefully observed to obtain the following 
kinds of data: 


Which objectives of the training were not accomplished (as evidenced by the 
trainee's performance on the mastery examination) ? 


Which items give the students undue difficulty? What were the errors the 
trainees made ? 


2. Validation Testing (or Field Performance Testing) in which the materials are considered 
to be ready for large scale use. A group of 15 to 30 students, representative of the 
total intended population, is selected. They are given the materials under the conditions 


in which they will eventually be used. Quantitative data on how well the materials 
perform is collected. 


In this unit we will consider the procedures and conditions under which the most information 
can be obtained from Developmental Testing. 


PANEL A 


0 ON O&O OO FF WON — 


Two adults - employees of the client company - have been asked to act as tryout 
students for the first draft of a program on electronic data processing. They report to 
the tryout room at the proper time. 

"Come on in," says the programmer, inviting them into the office he shares with 
two other members of the programming team for the computer project. "I'm Walt Burns, . 
and this is Stan Lovett and Dave Miller, the other programmers on the project." 

After a brief exchange of greetings, Walt invites them to sit down at a vacant 
eight foot table against one wall of the room. He slides up a chair and begins 
briefing them. 

"What we want you to do is take this program we're writing for your company . 
We've been working on it for a couple of months now and want to try it out on the 
persons who'll actually be using it. That's where you guys come in." 

"Now, here's the program." He hands the students copies of the program and 


several sharpened pencils. "If you have any questions, I'll be at my desk over there." 


ye 


CHECK YOUR EVALUATION AGAINST 
THE POINTS BELOW 


- 


Line 1. "Two adults - employees of the client company - have been asked to act as 
tryout students..." Good Tryout Technique. The tryout students should always be 
members of the intended population -- that is, the population who will eventually be 


learning from the materials. 


Line 1. "Two adults..." The number of tryout students used at any one time will vary 
from one to twelve. Beginning programmers cannot handle more than one at a time. 

More experienced programmers can handle not more than four at once. Usually, enough 
information is obtained from three students to make significant revisions in the material. 
This is especially true if the first three students are selected carefully. One should be 
chosen from among the most capable end of the continuum, another from the lower 

ability level, and a third from those with average ability. After revisions, a second 
group of three can be selected and put through the tryout procedure. 


Lines5 - 6. The informal atmosphere established by the programmer is good. Everything 
the programmer can do to create a cooperative feeling on the part of the tryout students 
will help, for he wants them to give him information. 


Lines7-8. "...sit down at a vacant eight-foot table..." Poor technique. The students 
should be situated so that they cannot see how far along the others are and so that any con- 
versation the programmer has with one will not tend to interrupt the others. 


Lines 7-9. Tryout room is an office in which other persons are working and where 
distractions are likely to occur. Poor technique. The tryout room should be used for 
that purpose only while the tryouts are taking place. 


Lines 10 - 13. Poor technique. The programmer has omitted several important items in 
orienting the students to the tryout situation. These are discussed in the checklist on the 
next page. 


Line 14. "I'll be at my desk over there." It's hard to tell from the description just where 


the programmer's desk is. But, if is important for him to locate himself where he can observe 
each student's answer and behavior, without distracting him. 


TURN TO THE CHECKLIST NOW. READ IT 


CAREFULLY, FOR IT IS BASIC TO AN UNDER- 
STANDING OF THE REST OF THE UNIT. 
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CHECKLIST FOR THE FIRST TRYOUT SESSION 


READ THE CHECKLIST NOW, BUT DO NOT ATTEMPT TO MEMORIZE ANY 
OF IT. YOU ARE EXPECTED TO BE FAMILIAR WITH ITS CONTENTS IN THE 
MATERIAL WHICH FOLLOWS. 


The programmer should first explain to the tryout student that the materials he is 
to be given are intended to help him learn subject matter designated in the title. 


The programmer should emphasize that the role of the student is to help the 
programmer evaluate some new educational materials. Comments and suggestions 
that the student makes will help the programmer make revisions. 


The programmer should then explain that he has to know how much the student 
already knows about the subject matter and whether or not the student has all 

of the prerequisites to learn from the materials. He should then give the student 
the pre-test (always) and the prerequisites test (if required)* timing the student 
on both. Both of these may be done when the test subjects are being selected. 


When the tests have been completed, the programmer should show the student 
the program and explain again that it is the material, not the student, that is 

to be tested from now on. This is an especially important point about which the 
student should have no question. 


The student should be given a ball point pen with which to write his answers. 
(This will prevent him from erasing potentially valuable information for revising 
the program.) He should be provided with answer sheets, if any. 


Tell the student to put an "X" next to the items he thinks he got wrong after he 
has checked his answer. If the program contains open-ended questions, tell 
the student about this. 


Explain to the student that if he doesn't know an answer, he should take a guess 
and write "guess" on the answer sheet. If he simply can't think of an answer, 
he should leave the answer blank and place an "X" next to the item on the 
answer sheet. 


Tell the student the time limits placed on the tryout session and that he can 
take a break whenever he feels like stopping. 


Re-emphasize that any comments he wants to write or express to the programmer 


will be useful and welcomed. 


10. Then ask the student to commence with the materials. (If the student asks what 
he should do or asks if he's doing it right, the programmer should gently insist 
that all the directions necessary are given in the materials. It is important to 
try out the directions, too.) 


11. The programmer should note carefully the time at the beginning and end of each 
tryout session and keep track of "break time”. 


*You give a prerequisites test if the program assumes skills such as mathematics or vocabulary 
knowledge that the students might not have. For example, a statistics program should have 
an arithmetic and algebra prerequisites test. 


You do not need a prerequisites test if the population can be assumed to have the required 


background. For example, management programs will not generally require a prerequisites 
test on company organization because a knowledge of this is assumed. 


ia, 
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WHEN TO INTERRUPT 


a a ee acca 


One of the difficult decisions to make during the tryouts is whether or not to interrupt the 
student's train of thought in order to find out what is giving him difficulty. Examine the 
situations in PANEL B and the ANSWER TO YOURSELF the questions below. 


Fe | ae SSS SS 


1. Which programmer got the most information for revision, the one in Situation A or the 
one in Situation B? 


A got the most 


B got the most 


on 


2. Could the programmer in Situation B have waited until the end of the tryout session to 
ask about the student's difficulty ? 


Yes 


No 


a 


3. Would the programmer have gotten as much information if he'd waited until the end of 
the tryout session? 

Yes 

No 


— 


a er en re Eee SEE: 


YOU'LL FIND SOME OPINIONS ON THESE QUESTIONS ON THE NEXT PAGE. 
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THE PROBLEM OF INADVERTENT TEACHING 


The purpose of the tryout is to get information from the student for revising the program. 
Beginners often forget this and waste valuable opportunities to listen to what a student has 
to say. Their enthusiasm for teaching or for the subject matter impels them to give 
information to the student instead of asking for it from the student. They teach a lot 
inadvertently -- even in seemingly innocent situations. (Note: "Teaching" may include 
any kind of "running commentary" from which the student can get cues.) 


EXAMINE THE DIALOGUE BELOW. 
ON YOUR ANSWER SHEET WRITE 
DOWN THE NUMBER(S) OF THE 


STATEMENTS WHICH EITHER DEFI- 
NITELY SHOW THE PROGRAMMER 
TEACHING OR WHICH SHOW THAT 

HE MAY BE TEACHING INADVERTENTLY. 


1. Programmer: | noticed that you raised your hand. 


Student: Yes, | can't seem to remember how to do it. Number 24. 


2. Programmer: Do you remember the answer you wrote for 22? 


Student: No...let me see. 


3. Programmer: Try reading the item again carefully. 


Student: O. K. 


4, Programmer: Think you can do it now? 


Student; Let me try it. 


5. Programmer: Good...you're doing fine...ooops, careful there...ah....fine. 
Go on to the next one now. 


2155 


ANSWER TO PREVIOUS PAGE: 


Statement lis O.K. In Statement 2 the programmer tells the student (or at least hints) 
where to find part (or all) of the answer. It's definitely teaching, as are Statements 3, 
4, and 5, in which the programmer first tells the student what to do and then gives a 
running commentary on how well he is doing. Instead, the programmer should have 
attempted to draw out of the student exactly what difficulty he was having. 
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a 
EXAMINE THE DIALOGUE ON THIS PAGE. CHECK THE STATEMENTS WHICH 


DEFINITELY SHOW THE PROGRAMMER TEACHING OR WHICH SHOW THAT HE 
MAY BE INADVERTENTLY TEACHING. 


a ee SSS 


Student: 1 don't think I'm getting this. 
Programmer: What seems to be the matter? 


Student: Well, I don't know exactly. | get this far (he points to frame 149) 
and then | don't know what to do next. 


Programmer: Let's see. You get as far as where it says "Place the decimal 
point and continue." on item 149, 


Student: Yeh. Well, not exactly. | guess it's before that. | didn't seem 
to understand back here in this part. 


Programmer: You mean here at 101? 
Student: Had trouble from there on. 
Programmer; Let's see if we can see what kind of trouble. 


Student: Well, | think | understand in general, you know? Like | know what 
|'m supposed to do. But | guess | just don't know how to round off right. 


Programmer: You think that you need more practice in rounding off? 
Student: Yes. There. Is that answer correct? 


Programmer: It'd help me more if you'd try to get the answer from the program, 
not from me. 


Student: (Glances at answer.) Guess | missed it again. 


Programmer: Let's see. You must have thought that you were supposed to 
take the square root of 23 and then go on fo this part. 


Student: No, but | understand now. I'm gonna try the next one. 


Programmer: O.K. Careful with setting up the problem. 


SZ 


ei Pe ee ee SS SSS SS aS 
ANSWER TO PREVIOUS PAGE: 


Statement 1 isall right. The programmer may be inadvertently teaching in Statements 2 and 8; 
it's hard to tell if he is being more specific about the place where the difficulty began than 

the student is. Statement 4 is fine. Statement 5 is a leading question. Instead of saying 
"You think you need more practice?" he should have said something like "What do you 

think would have helped you get the correct answer?" Statement 6 is excellent: part of 

what the programmer is testing in the tryout is how good the answers given in the program 

are. It is bad practice to tell the student whether or not his answer is correct. He should 

get that from the program. Statements 7 and 8 are both definitely teaching statements. 


ee ha a a ee ee 
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POST-TRYOUT INTERVIEW 


The following dialogue took place after the tryout was over. The programmer interviewed 
the student over special difficulties encountered in the program. 


Programmer: | noticed that you looked puzzled at frame 199. 

Student: Which one was that? | don't remember. 

Programmer: This one. (He opens the program to the frame.) 

Student: Oh, yeh. I remember that one. | couldn't figure that out for the longest time! 
Programmer: Let's look at it. First you took this number... 

Student: And then | put it... let me see... 

Programmer: Over here? 

Student: Yes. 

Programmer: And then didn't you divide? 


Student: Uh, huh. 


Check which of these is the best description of what went on in the dialogue above: 


The student told the programmer what his difficulty was and the 
programmer has valuable information for revision. 


___The programmer told the student what he did and hence did not 
find out for sure what went wrong because the student didn't 


tell him. He only agreed with him. 


Is it just as important for the programmer to be careful of inadvertent teaching after, as well 
as during, the tryout? 
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MAKING NOTES OF STUDENT COMMENTS 


Here's an example of part of a programmer's log of the tryout session. Examine it carefully. 


9:35 A.M. JR: 3835 
BS: 32 
VW: 25 


Programmer's Notes: 


_| Which frame each student is on 
every five minutes. 


3 parts of a note ona 
student-programmer interaction. 


IR 
Frame 36 


1. Initials of student 
2. Frame number 
3. Exact comment of student 


Programmer's note to himself 
as to what to change. 


"Are we supposed to round off?" 


Add to instructions. 


VW 
Frame 39 


flipped back to look at previous frame 


Note on non-verbal 
behavior of student. 
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Some Personal Observations by the Author 


| have found that persons with a background in clinical psychology have made the best 
program testers. The clinical attitude of encouraging information flow from the patient 
to the therapist is very analogous to the tryout situation. 


A useful procedure for training yourself to become sensitive to what you say during 
the tryout is to tape record all of your comments. Programmers with whom I've used this 
have found to their dismay that they teach a great deal more than they thought they did. 


One program we were testing was presented to students on white 4x 6 cards. |, as 
program editor, had made some rather severe remarks on pink cards of the same dimension 
and inserted them after the items on which the comments were made. In this particular 
tryout, the program got into the student's hands with the editorial comments inadvertently 
left in. (An inexperienced program tester who had not been told about the pink cards 
was doing the testing.) The student, an eighth grader, was delighted, and the amount 
of information he gave us was astounding. He got into the "swing of things" and was 
‘no longer bashful about telling us what was wrong with the program. To a comment of 
mine, "too easy," he added, "That's right." To another, "too repetitious," he wrote, 
"| needed the review." Now, from time to time, we use this technique to encourage 
student comment. 


Another way of eliciting student attitude is to ask the student to comment after each 
item (or sequence) using a probe such as the following: 


| think this item (sequence) is 


too hard 

too easy 
trivial 

too repetitious 


confusing 


other (please explain) 


a9]= 


DEVELOPMENTAL TESTING EXERCISES 


Take the tape marked "Student Tryout" and put it on your tape recorder. 
Do not start the tape yet. 


CHECK THESE POINTS ON YOUR TAPE RECORDER: 


1. The speed is set at 3-3/4 inches per second. 
2. The machine is set for playback. 


EXERCISE | 


In this exercise you will listen to the beginning of a student tryout. You will hear the 
programmer and a student talking. Your job during this exercise is to use the checklist 
( pages 6-7 ) to see if the programmer does all of the things he is supposed to do. Make 
a note of anything he omits or any questionable practice he uses. 


| 


TURN ON THE TAPE RECORDER NOW 


DO NOT TURN THE PAGE UNTIL THE 


TAPED PORTION OF EXERCISE | IS 
COMPLETED, 
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COMMENT ON EXERCISE | 


You will have found that the developmental test situation you heard in Exercise | was 
generally a good one. The programmer covered all of the points on the checklist. 


However, there are several subtle errors committed by the programmer which might 
possibly cause trouble. They are: 


The comment "...before we print 50,000 copies." The remark was 
made facetiously by the programmer, but if taken seriously 
by the student, it might influence his reaction to the program. 


The comments "...no sense your taking the program if you do well on 
the pre-test" and the enthusiastic "that's good that you don't 
know (the material already)." These might result in a paid 
subject deliberately doing poorly on the pre-test in order to 
qualify as a student for the program. 


Telling the learner the suggested average length of time to 
complete the program. If the student is at all sensitive about 
being average or above, he may hurry and thereby be something 
less than an ideal test subject. 
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EXERCISE If 


In this exercise you will again listen to a student tryout taking place. You are to do 
two things in the following exercise: 


1. take notes on the programmer-student interaction so that you 
can revise the program, and 


2. make note of anything the programmer on the tape does wrong 
during the tryout. 


For this exercise, you should divide a blank sheet of paper into two columns, heading 


one "revision" and the second "critique." This will assist you in listening for both 
types of problems and in keeping your comments straight. 


NOW, TURN TO PANEL C. 
This is a copy of the program the student is taking during the tryout. 


TAKE A FEW MINUTES TO FAMILIARIZE YOURSELF WITH THE PROGRAM. 


WHEN YOU HAVE FINISHED EACH STEP ON 
THIS PAGE, TURN TO THE NEXT PAGE. 
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Here's a picture of the tryout taking place. 
NOW START THE TAPE RECORDER. 


YOU MAY STOP THE RECORDER AT ANY POINT IN ORDER TO MAKE YOUR NOTES. 


[DO NOT TURN THE PAGE UNTIL THE VOICE ON THE TAPE TELLS YOU TO. | 


-25- 


ee, STUDENT DATA SHEET 


Age / } Date of Birth ‘A 7 na / § ) / IAG} Sex /¥) 


EDUCATION: Circle last year of schooling completed. 


12 3 4 5 6 7 8 9 0 NW @2> 13 4 «15 16 


TYPE OF HIGH SCHOOL COURSE: Circle. 


Academic Goechnical > Commercial Other 


ww COLLEGE MAJOR DATE GRADUATED 
COLLEGE MINOR 


OTHER SCHOOLING (Please specify) 


EMPLOYMENT: List your last three jobs, the companies you worked for, and the dates of 


your employment. H ; 
Just grades tit) 
f 4 


WHEN YOU'VE FINISHED EXAMINING 

THE DATA SHEET, TURN ON THE RECORDER. 
DO NOT TURN THE PAGE UNTIL YOU 

ARE TOLD TO DO SO BY THE TAPE. 
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THE LETTER BELOW IS THE PRE-TEST FOR THE PROGRAM ON BUSINESS ENGLISH. 
EXAMINE THE PRE-TEST. 


PRE-TEST FOR PART | 


—————— ee 


Correct the following letter by crossing out the incorrect capitalization and inserting the 
correct, 


a 


JONES AND COMPANY 
195 West 34th Street, New York 94, N.Y. 


January 15, 1964 
he 
ee o. weeks 
15 Beck street 
Los Angeles, California 


Dear Mr. weeks: 


Thank you_very much for your letter of december 29 from san francisco. 
After reading it ¥ immesiately checked on when the queen mary would sail. 
Then jlwalked over fo 'penn station to see wher’ jane's train would arrive. 
| met her at the station and we walked for awhilein the Winter snow. New 
year's day fell on a wednesday this year. The celebration here was wild. 
I'm sure’ jane enjoyed it. 


You'll let me know, | hope, when you'll be in new york next. Let's have 
lunch together. 


Sincerely, 


Alexander Jones 
Vice President 


WHEN YOU'VE FINISHED EXAMINING 
THE PRE-TEST, TURN ON THE TAPE 
RECORDER AGAIN, 


DO NOT TURN THE PAGE UNTIL YOU 
ARE TOLD TO DO SO BY THE TAPE. 


207% 


C 


Ge art Arey > 


hel Auman 


- { 
Le ger 
/ 


te net 
DI web Tr aw Sako? 


es Sadiune 
, | 


Vv Ne 
~~ Uer 
ie Chea 
CeO NeW 


| Orme wt vu vow! 


/ CEL Thewe 


a aii (pr CAEN VUONVH x 


4 ‘t AW SSUE VA ALE LG YU 


i. pire Sing QArks ew A 


— Cok uc Tall 
2 Geuntye Lear 


wy Le Pes thre Kanes aby 


tl earl: 
hed. ee CAL oo? (ae “y wal: uf art 
Bvaee es | 
DO NOT TURN THE PAGE 
o UNTIL YOU ARE TOLD TO 
ad DO SO BY THE TAPE. 
eeu SOG 
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THE LETTER BELOW IS THE POST-TEST FOR THE PROGRAM ON BUSINESS ENGLISH. 
EXAMINE THE POST-TEST. 


POST-TEST FOR PART | 


Correct the following letter by crossing out the incorrect capitalization and inserting the 
correct. 


JONES AND COMPANY 
195 West 34th Street, New York 94, N.Y. 


January 15, 1965 


rA Ch 
Mr. yalph o. decks 
15 Beck }treet 
Los Angeles, California 


Ww 
Dear Mr. #veeks: 
D 
Thank you very much for your letter of december 29 fi a francisco. 
After reading it 7 ‘immesiiately checked on ye the queen ‘mary would sail. 
Then #-walked over to 'genn$tation to see wher Zane's train would arrive. 
a met her at the station and we walked for a while in the Winter snow. New 
Y year' s oy fell on a Wednesday this year. The celebration here was wild. 
I'm sureYjane enjoyed it. 


You'll let me know, | hope, when you'll be in Wew york next. Let's 
have lunch together. 


Sincerely, 


Alexander Jones 
Vice President 


LOOK OVER YOUR COMMENTS FROM THE TAPE AND ADD ANY YOU MAY HAVE | 
RESERVED TO THE END OF THE TAPE. WHEN YOU'VE FINISHED, TURN TO THE 
NEXT PAGE. 
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wy) wo 
COMPARE THE NOTES AND COMMENTS YOU MADE WITH THE ONES BELOW 


a 


These are the major mistakes the programmer made: 


1. The room in which the tryout takes place is not sufficiently isolated 
so as not to distract the student during the tryout. 


2. The tape implies that the programmer gave the student a pencil 
rather than a ball point pen. 


3. The programmer failed to repeat after the pre-test that the materials, 
not the student, were being tested. The student was not given suffi- 
cient orientation about each phase of the tryout, especially with 
regard to the purpose of his correcting the letter (which is labeled 
"pre-test"). 
4, The student was not told to put "X"s beside mistakes. 
5. The student was not told to guess, when he couldn't figure out something. 
eed 6. The student was not urged to make comments or criticisms on the program. 
7. No time limits for the tryout session were explained to the student. 


8. The student was not told that he could take a break whenever he wanted to. 


9.  Atone point in the course of the tryout, the programmer told the student 
where to look for an answer, thus inadvertently teaching him. 
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wo 


The following are the notes you should have made during the tryout: 


FRAME NUMBER 
STUDENT'S INITIALS 


Pre-test 
E.G. 


"not enough room to put in the big letters" 


3 
Ev. 
"Do | turn the page?" 


4 
E.G. 
"What am | supposed to do here?" "Put a check mark." "It isn't on my answer sheet." 


1 
E.G. 


student looking back in book, says "trying to decide if Robert is a common or a proper." 


(We didn't make a note of the student's behavior on item number 13 because the student 
put "X" on his answer sheet next to the question.) 


E.G. 
student got the wrong answer, but he did compare his answer to the answer sheet. 


17 
E.G. 
student spent a long time on this one 


19 
E.G. 
"Do | have to write whole sentences?" 


21 


E.G, 
puzzled a long time, then asked: "I don't know what lower case means." 
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POST-TEST ON DEVELOPMENTAL TESTING 


DIRECTIONS 


Prepare a checklist or outline of the important things to do and not to do 
in developmental testing. 


Write the checklist as if you were making notes to explain the procedure 
to someone else. Be sure to include what you would do before, during, 
and after the tryout. 


Prepare your outline on separate sheets of paper. 


At the top of each sheet, write your name and today's date. 
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DEVELOPMENTAL TEST 


Answers to Post-Test and Self-Test 


Population 


should be members of intended (or target) population 


- number of tryout students (3-12) 


should range from lowest ability level to highest in population 


- no more than 3 students for one programmer during tryout sessions 


Room 
- informal atmosphere 
- free from distractions 


~ keep students separated so they cannot see what others are doing 


Equipment 

- programmed unit to be tested 

~ answer sheets 

~ ball point pens 

- pre-tests, prerequisites tests, and post-tests 
- attitude questionnaires 

- student personal information forms 


- programmer's report of student tryout 


Instructions prior to program 

- materials, not student, are being tested 

- materials are experimental - students are helping programmer find mistakes 

- need to take prerequisities test to see if he knows everything he needs to in 
order to take the program 


- need to take pre-test to see how much he already knows 
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- explain that Pl materials are self-instructional and that he can proceed at 
his own speed 


- ask him to place 'x' next to answers he got wrong 
- urge him to make comments for answer sheet 
- ask him to guess, if he can't think of answer 


- explain time limits 


tell him he can take a break when he feels like it 


explain that the rest of the directions are given in the material 


Programmer's job 


be thoroughly acquainted with the unit being tested 

- note time for beginning and end of each session and any breaks taken 

- note what frame student is on every five minutes 

- observe student for signs of puzzlement, boredom, fatigue and note these 


- take advantage of opportunities to encourage student to comment - be careful 
not to discourage student 


interrupt student only when he can't proceed on his own 


if interrupting student, try to alter material on the spot 


if on-the-spot alterations fail, explore the difficulty with students using 
open-ended questions 


- be on the alert not to inadvertently teach the student while exploring a 
difficulty with him 


- interview the student about all difficulties at end of tryout session 


- make careful notes of exact words of student, student's identity, and item 
number of program 


\R\ Copy 


developmental testing 
panel book 


PANEL BOOK 


DEVELOPMENTAL TESTING 
Trying Out Programmed Instructional Materials 


With Individual Students 


Robert E. Horn 


Copyright 1964, Robert E. Horn 


PANEL C 


CONTEMPORARY BUSINESS ENGLISH 


Part | 


Using Capital Letters Correctly 


PRETEST FOR PART | 


CORRECT THE FOLLOWING LETTER BY CROSSING OUT THE INCORRECT CAPITALIZATION 
AND INSERTING THE CORRECT. 


JONES AND COMPANY 
195 West 34th Street, New York 94, N. Y. 


January 15, 1965 


Mr. ralph o. weeks 
15 Beck street 
Los Angeles, California 


Dear Mr. weeks: 


Thank you very much for your letter of december 29 from san 
francisco. After reading it i immediately checked on when the 
queen mary would sail. Then i walked over to penn station to see 
when jane's train would arrive. | met her at the station and we 
walked for a while in the Winter snow. New year's day fell ona 
wednesday this year. The celebration here was wild. |'m sure 
jane enjoyed it. 


You'll let me know, | hope, when you'll be in new york next. 
Pp Y Y 
Let's have lunch together. 


Sincerely, 


Alexander Jones 
Vice President 


1. 


Proper names are the names of particular people, places, 
or things. 


A. the spaceman 
B. John Glenn 


Which of these words names a particular person? 


John Glenn 


A. the ball park 
B. Yankee Stadium 


hich of these names a particular place? 


Yankee Stadium 


3. 


Proper names are the names of particular people, places, 
or things. 


A. the girl 
B. Jane 


the girl 


Which of these is not a proper name? 


—— ae 


"Chicago" (__is__ is not) a proper name. 


Ds 


The first letter of a proper name should always be a 
capital letter. 


| went to see sandra. 
Copy the sentence below and capitalize the 


proper name in it. | went to see 
Sandra. 


EXAMPLE: Who is your girl friend? 


Is there a proper name in this example? 


A. Who is your girl friend? 
B. Her name is Sandra Nelson 


Both of these sentences are properly capitalized. 


Is there a proper name in sentence B? Yes 


No 


(how many ?) words in the 


Is the first letter of each word in her name 
capitalized? Yes 


No 


EXAMPLE: His name is robert frederick doe. 
opy the example below. (Be sure to capitalize it 


orrectly .) His name is 
Robert Frederick Doe. 


A. I'm from New york. 
B. I'm from New York. 


Which sentence is correct? 


10. 
Nouns which are not proper names are called common nouns. 
EXAMPLE: 1 saw the man. 


The word "man" in the example is ac 


EXAMPLE: | saw Robert. 


In this sentence, Robert is a ( common noun 
proper name). 


A. alice 
B. new york 
C. ralph 


hich of these words should be capitalized? 


none of them 
only A 

only B 

only C 

both A and C 
all of them 


proper name 


all of them 


A. What ship are you sailing on? 


B. ['m going on the Independence. 
The word ship in sentence Aisa (common noun 
proper name.) common noun 


The word Independence in sentence B isa 
(| common noun proper name). proper name 


A. What is that building? 
B. It's the Empire State Building. 


Both sentences are properly capitalized. 


hat is a general rule that you can give about the 
capitalization in sentence B? 
Each word of a 
proper name is 
capitalized. 
(or equivalent 
response) 


EXAMPLE: Could you tell me where the 
kings county building is? 


Copy the example below and capitalize each letter 
lof the proper name. Could you tell me 
where the Kings 
County Building is? 


A. | went to california last year. 


B. I've lived in thirteen states. 


Copy the sentence containing the proper name 
below and capitalize the proper name. | went to California 


last year. 


A. collins college 
B. the university 


hich example above is a common noun? 
A 


B 


JIM: Have you ever been to europe? 
BILL: No. I've lived in the united states 
all my life. 


Europe 
United States 


1? 


The word "I" is always capitalized, Which of the 
following sentences is correct? 


A. Then i went to the movies, 
B. After that | went home. 


0. 


ommon nouns ( _—_are always are never) 


apitalized. are never 


| 
EXAMPLE: Do you think j/should go? 


There are two words in the sentence below which need 
apitals. Cross out the lower case letters and place 
capital letters above them where capital letters 


are necessary. 


L A 
| think you should go to los angeles. .. Jos dngeles... 


22. 


In the example below, cross out the lower case letters 
and write capital letters above them where necessary . 


First i went to the drugstore. 


23. 


Days of the week, months of the year, and the names 
of holidays are capitalized. Check which of the 
following should be capitalized. 


monday months 


january november 
days birthday 


christmas holiday 


A. When were you born? 
B. {| was born on thursday, april 19, 1940. 
There are two words in sentence B which do not have 


capital letters and should have them. Cross out the 
letters and write the correct capital letter above them. 


| 
First 7 went... 


monday 
january 
christmas 


november 


T A 
.. #hursday, April 


A. What day does easter come on this year? 
B. It comes on sunday every year, stupid. 


Cross out the mistake(s) and write the correct 
capital letter above the word. 


How many holidays are there in february ? 


. Two. We get off from school on 
Washington's Birthday and lincoln's 
birthday. 


hich three words in these two sentences should be 
Write them correctly below: 


February 
Lincoln's 


Birthday 


2h. 


Place a check mark beside each of the common nouns. 


A. man __D. thanksgiving day 


B. florida aie Ee month 
C. holiday ___F. july 


A. washington street 
B. the street 


C. a beautiful street 


One of these examples needs two capital letters. 


Write it correctly below. 


Why does it need capital letters? 


EXAMPLE: I live at 319 oak street. 


Copy this sentence below and put in the correct 


A, summer 
B. July 


hich of these words is the name of a season 


Washington Street 


It needs capital 
letters because 

it is the name of 
a particular place. 


] live at 319 
Oak Street. 


31. 


The names of the seasons are not capitalized. 


June, July, and August are the (| _summer 
Summed) months. ____ summer 
EXAMPLE: Fall is another name for autumn. 
Is this sentence correctly capitalized? 
___Yes 
No ___ Yes 


—— 


33. 


Now, here's a review of some of the things you've learned 


labout capitalization. Correct the sentences below: 


A. then sally and i went to boston. Then Sally and I 
went to Boston. 


does martha live on madison avenue? : 
Does Martha live 
on Madison Avenue ? 


B. 


DI Vi helmOeP 9M BaN shaw. Loh Sle eNeG 


a self-instructional program by 
— Robert E. Horn* 


Here is an excerpt from a hypothetical individual testing session from the "Developmental Testing" 
program. The student is having difficulty with an item. 


Programmer: Let's see. You must have thought that you were supposed to take the 
square root of 23 and then go on to this part. 


Student: No, but | understand now. I'm gonna try the next one. 


Programmer: O.K. Careful with setting up the problem. 
This programmer has made two errors: 


1. in his first statement, he speculates on the problem rather than asking the student 
what was causing the difficulty, and 


2. in his second statement, he is inadvertently teaching, which will prevent the testing 
session from being a reliable measurement of how well the program can teach. 


When the program is group tested, students will probably make a number of errors. As often happens, 
the programmer may decide that individual testing is not useful. But it is his incomplete knowledge 
of how to individually test that keeps him from obtaining useful information for revision. 


~ CONTENT: The "Developmental Testing" program teaches: 


.. establishing a suitable environment for the testing session, 
. .what instructions to give to the test subject, 
. when and how to intervene in the testing process, 
. encouraging useful comments from the student, 
. avoiding inadvertent teaching, and 
. taking notes on student comments and behavior. 


FORMAT: The program package includes the program book, a tape recording of good and bad 
individual testing sessions, and a panel book showing the program being tested on 
the recording. 


COST: The prices listed below include the program book, the panel book, and the tape. 
1 - 6 — $8.95 each 
7 or more — $6.95 each 


*Mr. Horn is a Research Associate at the Institute of Educational Technology, Teachers College, 
Columbia University . 


Detach and send to the Center for Programmed Leaming for Business, 340 South State Street, 


Please send _ copies of "Developmental Testing" to: 
Name 
. Title 
“ae Organization 
Street 
City State Zip Code 


Check enclosed Bill my company 
84-1 


DESCRIPTION OF 
A SELF-INSTRUCTIONAL PROGRAM FOR TRAINING PROGRAMERS IN 


DEVELOPMENTAL TESTING (STUDENT TRYOUTS) 


The accompanying chart shows the teaching strategy for a 
self-instructional program for training programers in developmental 
testing (or student tryouts). 


. In the first phase the trainee programer is taught 
certain aspects of how to conduct a developmental 


test or student tryout of a programed instructional 
sequence; major emphasis is placed on the programer- 
student interaction, 


. In the second phase the trainee programer listens to 
the programer-student interaction on a tape recorder, 
He has in front of him at various points in the listening 
process the program which the student is taking, the 


pre- and posttests, the student's personal data sheet, 
and the student's answer sheets. 


. The trainee programer's job in this second phase is to: 


. Take notes as if he were the programer in 
the developmental test situation on the 
tape recorder. 


. Criticize the programer's testing technique, 


. In the third phase the trainee programer conducts a 
developmental test of a program he has written with 
a student. He records his interaction with the 
student. 


. In the fourth phase he criticizes his own interaction 
with the student, thus providing himself with feedback 


on his testing technique, 


This program has been in operational use for the past year and 

a half in various versions, It was developed in the author's 
beginning and advanced workshops in programed instruction while 

he was Director of Training and Consulting Services for the Center 
for Programed Instruction, Inc. It has also been used by a company 
whose major product is programed instruction for training its 
propramers, 


AN INSTRUCTIONAL SYSTEMS DESIGN BY ROBERT E. HORN 


SELF-INSTRUCTIONAL PROGRAM FOR TRAINING PROGRAMERS IN 
DEVELOPMENTAL TESTING (STUDENT TRYOUTS) 


TRAINEE on 
Seer PROGRAMER PANEL FRAME TAPE ee 
BOOK BOOK RECORDER 
FRAME 
BOOK. Trainee Con- Trainee 
—— ducts Devel- criticizes 
opmental Test his 
of PI he has interaction 
written. with 
TOPICS TAUGHT CONTAINS CONTAINS CONTAINS He tape re- | |begoent 
cords his 
- Overview of . Program on .- Student . Tape interaction maou 
Developmental Business Data recording with student. ) 
Testing English Sheet of 
- Encouraging . Pretest . Student interaction 
Students to Answer between 
Make Comments - Postest Sheets programer 
.- When to , and 
Interrupt ee of student 
- How to Intervene and Posttest 


- The Problem of ED 


Inadvertant Teaching 
. Making Notes of TRAINEE PROGRAMER'S BEHAVIOR: 


Student Comments . Take notes as if you were the 
programer on the tape recording 


. Criticize the programer's Testing 
technique 


AN INSTRUCTIONAL SYSTEMS DESIGN BY ROBERT E. HORN 


NeW 
“POorams 


‘Development Testing,” by Robert 
E. Horn. Published and distributed 
by The Center For Programmed 
Learning For Business, The Uni- 
versity of Michigan, Ann Arbor, 
Mich. 1964. 35 pp. 8'2 x 11 in., 
soft cover. Panel book, 12 pp. Tape 
recording, 15 min., 334 ips. $8.95; 
quantity discount. 


‘Developmental Testing” is for the 
beginning writer of programed in- 
struction. The author of the pro- 
gram, Robert E. Horn, comes well 
qualified for his task. He was for- 
merly Director of Training and Con- 
sulting Services for The Center For 
Programed Instruction. The pro- 
gram grew out of his practical 
experience from directing workshops 
in programed instruction at the In- 
stitute of Educational Technology, 
Teachers College, Columbia Uni- 
versity. 


(ting feedback is difficult 


“He found that one of the most 
difficult lessons for the beginning 
program writer is how to be sen- 
sitive to the feedback he gets from 
subjects during early tryouts of a 
program. So, he analyzed the task 
and programed it. 

By “developmental testing,” the 
author means individual learner try- 
out during the writing process to get 
qualitative information for revising 
the program. This has nothing to do 
with validation testing, the field- 
performance tryout of the program 
by groups to confirm that it meets 
its stated objectives. 

The program is attractively pack- 
aged. It consists of three simple 
parts: 

(1) Workbook with exercises. 

(2) Tape recording of simulated 

testing sessions. 

(3) Panel book showing the les- 

son being tested on the tape. 


Key topics of the program stress 
F Tto: 
xt) Set up a suitable test environ- 
ment. 
(2) Give instructions to the test 
trainee. 
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(3) Intervene in the testing 
process. 

(4) Encourage useful comments. 

(5) Avoid inadvertent teaching. 

(6) Take notes on learner com- 

ments and behavior. 

“Developmental Testing” is an ac- 
tion project designed for self-teach- 
ing. It assumes that the learner is 
acquainted with the use of programs 
and that he has already been intro- 
duced to the writing of programed 
instruction. 

To get an idea of what the pro- 
gram is all about, you might like to 
try and answer some questions from 
the pretest. On a separate sheet, pre- 
pare a simple outline, as if you ex- 
plained the job to someone else, of 
the important things to do and not 
to do in individual tryout. Use com- 
mon sense: 

(1) How many tryout trainees? 
(Subjects) What kind? What should 
be their characteristics? How many 
at one session? 

(2) How do you arrange your 
tryout room? (Environment) 

(3) What tryout materials will 
you use? 

(4) How do you brief the tryout 
trainee? On P.I.? On the test pro- 
gram? On his testing job? 

(5) What else must you do, as a 
program writer, before, during and 
after tryout? What do you prepare, 
observe, record? 

For suggested answers, see end of 
this review. 


How fo use the program 


Here’s how you use the program: 
First, you study typical episodes dur- 
ing tryout and follow the dialog be- 
tween programer and trainee. Then, 
you practice how to differentiate be- 
tween correct and incorrect methods. 
You make notes about mistakes and 
omissions and point out techniques 
which could be improved. 

Finally, you listen to the audio 
tape on a standard tape recorder. 
You hear typical tryout sessions and 
follow the interchange between pro- 
gramer and trainee. With the help 
of the panel book, you critique the 
interaction and make notes to sug- 
gest corrections in the test program. 

To sample the flavor of the tape, 
let’s take two typical excerpts. The 
first is an example of poor technique, 
the second is a recommended ap- 
proach. 

(1) Traiee: “I’m having difficulty. 


PROGRAMER: “Which one? 

T: “Number seventy-five. 

P: “Oh, seventy-five! The two guys 
last week had trouble with that one 
too. You didn’t understand what 
‘parameter’ meant. 

T: “Well, Pm not positive that I 
know. 

P: “It means a ‘limit-—as the limit 
of a problem or a job.” 

(2) TRAINEE: “I’m confused. 
PROGRAMER: “On which frame? 

T: “Both of these—one twenty-four 
and five. 

P: “Let’s'see. 

T: “I got the first, but not the sec- 
ond. 

P: “Hmmm. I think I know what 
may be giving you trouble. Suppose 
I change this sentence this way. Does 
that make any difference? 

T: “Yes. It’s perfectly clear now. 

P: “Ill just write those new words 
right in the frame. It'll help the next 
person who does the program.” 

The tape recording has some tech- 
nical flaws: unintended repetitions, 
some sound overlap and echo ef- 
fects. The voices do not sound pro- 
fessional. However, the episodes do 
come across and simulation is well 
done. 

Instructions on the tape are clear 
and apparently well tested, a point 
often overlooked by busy program- 
ers. Sound signals serve to alert you 
to action. Pacing is reasonable. You 
are invited to make notes while lis- 
tening, stop the tape whenever you 
like, rewind to listen over again. 
How long does it take? About 45 
minutes for the complete course. 
That includes 15 minutes of tape 
recording. 


Used at Michigan workshop 


The program is being used in the 
workshop for beginning programers 
at the University of Michigan to 
supplement training in developmen- 
tal testing. Immediately following 
the program, learners conduct try- 
out interviews to practice their newly 
gained skill. 

Through observation, the work- 
shop staff found that the program 
teaches these important skills: 

(1) How to sift out relevant 
trainee feedback. 

(2) When to shut up and listen. 

(3) How to avoid overlistening 
to trainee criticism. (‘““You don’t 
like it? All right, I'll change it.”) 

(continued on page 42) 
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Only the 


PROGRAMMED | 


INSTRUCTOR 
combines 
motion pictures 
still pictures 
sound 
questions - 
answers and 
review 

in ONE 
automated 
A-V training 
system. 


With this system, the trainee works at 
his own speed . . . stops at a question 
or a diagram . . . reverses to review a 
motion sequence of a field problem... 
moves ahead to learn more. This stop- 
and-go learning experience . . . which 
uses sound, motion, and study pauses 

. improves subject retention and in- 
Creases learning speed 25 percent over 
other programmed systems. 


The Port-A-Film A/V programmed In- 
structor is used for individual or group 
instruction. It is automated . . . car- 
tridge loaded with coordinated sound 

. and can be programmed for any 
motion or still sequence. 


Write or phone for further Information 
or a demonstration. 


PORT-A-FILMS 
PRESENTATIONS, Inc. 


422 MADISON AVENUE, NEW YORK CITY 10017 
Telephone (212) 421-5811 


Manufacturers of the A/V Programmed Instructor 
Production Center for Programmed Films 
For Information, Circle 441 on Post Card 


PORTA FILM | 
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NEW PROGRAMS 
(continued from page 39) 


(4) How to avoid teaching the 
tryout trainee. (“You don’t get it? 
Let me tell you. . .””) 

Most learners at the Michigan 
workshop react favorably to the pro- 
gram. They like the simulated ses- 
sions between tryout trainee and 
programer. It gives them a feel for 
the tryout sessions. It makes them 
not only more receptive, but also 
more selective when listening to 
trainee comments. 


Shortcoming 


Perhaps the major weakness of 
this program is that it does not give 
the learner a definite statement of 
training objectives. As a result, the 
posttest merely asks the beginning 
programer to develop a_ checklist 
of the do’s and don’ts of develop- 
mental testing. In effect, it is the 
same as the pretest, a pencil-and- 
paper measure which serves merely 
to verbalize the job. Clearly, this is 
not an effective test of the skills the 
learner practiced in the program’s 
exercises. Why have him write about 
the job? He is supposed to do it. 
Obviously, if we just supply our bud- 
ding programer with a good check- 
list, he will have little need for 
developing his own, every time he 
does developmental testing. 

Instead, the posttest should focus 
on the desired skills: How to set up 
the tryout session, how to give in- 
structions, how to intervene, how to 
listen, how to avoid teaching and 
how to make notes. By writing out 
definite training objectives in terms 
of observable skills, a more mean- 
ingful posttest could be developed. 
For example, a role-play test can be 
designed to measure training out- 
comes. 


To summarize, “Developmental 


Testing” shows on the minus side: 
(1) No stated training objec- 
tives. 


(2) Posttest not relevant to de- 
sired training outcomes. 

(3) Nonprofessional 
with some technical flaws. 

It scores on the plus side with: 

(1) Challenging exercises and 
busy action projects. 

(2) Good simulation of tryout 
sessions. 

(3) Building and developing use- 
ful skills. 


recording 


What if the beginning programer 
does develop sensitivity to student 
feedback? What is he going to do 
with those critical comments? How 
does he revise his manuscript? That 
will be the subject of future pro- 
gramed exercises. “Developmental 
Testing,” the author explains, is 
only the first unit of a larger program 
now under development. 

It would be exciting to see the 
author in future program units 
achieve even greater job simulation, 
create more relevant posttests and 
design stronger follow-up exercises 
by exploring the opportunities of 
video tape. 

No validation data is as yet avail- 
able from the publisher. Hopefully, 
such data will be published when the 
entire program has been completed. 
But why hesitate to try a program 
because it offers no validation data? 
Consider this: reading a review or 
studying validation data can only 
help you preselect a program. Your 
final evaluation of any program must 
come from testing it on your own 
trainees. So, if you or your staff want 
to sharpen your skills in develop- 
mental testing, why not try Robert 
Horn’s “Developmental Testing” 
program? 


Suggested answers for self-test 


(1) Use 3 to 12 tryout trainees; 
no more than 3 per session. Use 
members of intended audience, rep- 
resenting entire ability range. 

(2) Keep room informal, free 
from distractions; trainees separated. 

(3) Have ready: lessons, panels 
exercise materials, pencils, answer 
sheets, tests, opinionaires, trainee 
data sheet. Also, specified study 
aids: dictionary, slide rule, micro- 
scope, etc. 

(4) P.I. is self-instructional, self- 
pacing. Experimental material, not 
trainee, is being tested. Take pre- 
test. Help find mistakes as co-editor. 
Make comments. Find all other di- 
rections in the lesson. 

(5) Know lesson thoroughly. 
Keep time record. Observe trainee, 
check his progress regularly; note 
fatigue, boredom, puzzlement. Inter- 
rupt only when he is stuck and try 
to correct material on the spot. En- 
courage frank comments. Explore 
difficulties with open-ended ques- 
tions. Avoid teaching. Avoid over- 
listening to criticism. Keep trainee 
records.—RALPH A. DREKMANN 
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